test: add StartableToolSet backoff regression tests and docs - #4065

Review: needs work before this can land

COMMENT only — not approvable yet. This PR is still a draft, Go CI has not run for this stacked branch, and the test/docs changes are out of sync with the current #4062 implementation.

The test approach has genuine value—especially the consumer-shaped fakes, synctest coverage, and real rag.New integration coverage—but it needs to be rebased and retargeted before it can validate #4060’s production path.

CI status: Go CI did not run

Only documentation workflows ran and passed. This PR targets fix/startable-toolset-backoff, while the Go CI and CodeQL workflows trigger only for pull requests targeting main.

Therefore, build-and-test, lint, Windows tests, and CodeQL did not execute for these new Go tests. The PR should be rebased/retargeted as appropriate and receive full Go CI before approval.

[blocking] The stack is stale; five tests fail against the current #4062 head

This PR is based on the first #4062 commit (5974d12), while #4062 has since advanced through 072cc4a and e74816b.

Those changes altered the exact contracts encoded here:

Rebasing onto the current #4062 implementation produces failures in:

• TestBackoffRegression_RAGShapedFailureAndRecovery
• TestBackoffRegression_HTTP408EngagesGate
• TestBackoffRegression_ConcurrentStartsNoMultiplication
• TestBackoffRegression_JitterDeSynchronizesRetries
• TestRAGStartableBackoff_StatusErrorEngagesGate

Please rebase this PR before refining the test expectations. The failures are behavioral contract drift, not build or lint failures.

[blocking] The tests exercise Start(), not the production retry path

The retry storm in #4060 occurs through the non-blocking per-turn startup path:

• pkg/agent/agent.go → TryStartWithTimeout
• pkg/runtime/runtime.go → TryStartWithTimeout

Current #4062 deliberately keeps blocking Start() immediate; the retry gate applies through TryStart / TryStartWithTimeout.

However, the gating scenarios in this PR call Start() directly, for example in:

• pkg/tools/startable_backoff_regression_test.go
• pkg/tools/builtin/rag/rag_backoff_test.go

After rebasing, those assertions either fail outright or pass without testing the intended gate. Please:

1. Move positive gate assertions to TryStart / TryStartWithTimeout.
2. Add an explicit inverse test proving that blocking Start() remains immediate and bypasses automatic backoff.
3. Keep the concurrency test, but drive concurrent TryStart calls at the retry boundary so it verifies the real single-flight behavior.

This is important: otherwise the suite does not protect the code path that produced the incident.

[blocking] Delay and jitter documentation is materially incorrect

docs/tools/rag/index.md currently describes:

• base delay: 1 second
• maximum delay: 30 seconds
• equal jitter: [nominal/2, nominal]

Current #4062 implements:

• base delay: 15 seconds
• maximum delay: 5 minutes
• additive jitter: [nominal, 1.2 × nominal]

The existing docs understate the possible delay by an order of magnitude and describe the wrong jitter model. Please update the RAG documentation and test arithmetic together, preferably using exported constants in tests rather than duplicating literal bounds.

The current 31s sleep-based assertion in the RAG test must also be replaced with the injected retry clock; it is no longer beyond the maximum retry window.

[blocking] MCP/LSP documentation promises fail-fast behavior the current classifier does not guarantee

The RAG/MCP/LSP docs state that only structured HTTP errors from backing model providers trigger startup pacing and that local MCP/LSP failures fail fast.

Current #4062 delegates to modelerrors.RetryableHTTPStatus, which can extract 4xx/5xx-looking status values from error text through a regex fallback. As a result, errors such as these can enter the backoff path regardless of origin:

• remote MCP handshake failure containing 503
• local endpoint/process error containing a status-like 503
• LSP/process error containing an incidental 5xx-like number

Either:

1. narrow the classifier in fix(#4060): add bounded jittered backoff to StartableToolSet retry path #4062 to match the documented guarantee; or
2. revise the RAG, MCP, and LSP docs—and this PR’s compatibility tests—to accurately describe the broader behavior.

Please add a test for a non-model/local error string containing a status-shaped value. That would mechanically protect the intended contract whichever direction we choose.

[should-fix] Cover TryStartWithTimeout timeout behavior

Since this is intended as the regression suite for #4060, add focused coverage for the timeout path:

• an abandoned in-flight start must not schedule a second overlapping attempt;
• the retry window is armed only when the underlying attempt actually settles with a qualifying failure;
• cancellation/deadline errors do not themselves create a backoff window.

This is the most relevant lifecycle edge case for expensive RAG initialization.

[should-fix] Correct the recovery-warning claim

The RAG docs currently say a warning is logged again on recovery. The current success path intentionally suppresses recovery logging. Please either remove that statement or add the actual recovery signal in the implementation—preferably the former for this PR, to preserve scope.

[should-fix] Reduce duplicated coverage

Several scenarios overlap the base #4062 test suite: basic 429 gating, cancellation, retry boundary behavior, and jitter. The highest-value additions here are:

• RAG integration through the real error-wrapping chain;
• MCP/LSP compatibility cases;
• concurrent non-blocking attempts;
• no-timer/no-goroutine-leak behavior.

Consider consolidating duplicated generic cases into the #4062 suite so policy changes have one authoritative set of timing expectations.

[optional] Strengthen the jitter assertion

After updating to additive jitter, assert the full expected range for every sample ([d, 1.2d]) as well as meaningful variation. len(seen) > 1 alone is too weak to catch a number of broken jitter implementations.

Suggested path forward

1. Rebase onto the latest fix(#4060): add bounded jittered backoff to StartableToolSet retry path #4062 head, or retarget to main after fix(#4060): add bounded jittered backoff to StartableToolSet retry path #4062 merges.
2. Replace blocking Start() gate assertions with TryStart / TryStartWithTimeout.
3. Add an explicit test that blocking Start() remains ungated.
4. Derive test windows from the current exported base/max constants and injected clock.
5. Update docs to the actual 15s → 30s → 1m → 2m → 4m → 5m progression and additive jitter.
6. Reconcile MCP/LSP/RAG wording with the current status-extraction classifier.
7. Ensure full Go CI runs on the rebased/retargeted PR.

The structure is promising. Once it follows the current #4062 contract, this should provide valuable cross-layer regression coverage for #4060.

🤖 Automated implementer agent — this comment was posted by the implementer bot from Docker Agentic Platform, not by a human developer

Folded into #4062 — test and documentation commit cherry-picked directly onto the fix branch. All content from this PR (regression tests, RAG integration tests, and docs) is now at HEAD 501cab1 on fix/startable-toolset-backoff.